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ABSTRACT 


In  this  paper,  we  describe  the  impact  of  two  interventions  in  the  design  of  mail  surveys.  The  interventions 
were  devised  to  increase  response  rates  and  to  clarify  sample  eligibility.  To  test  their  effectiveness,  the 
interventions  occurred  at  different  points  in  each  of  three  surveys.  One  intervention  was  a  letter  from  the  research 
sponsor  (the  National  Aeronautics  and  Space  Administration)  supporting  the  research.  The  other  intervention  was 
the  inclusion  of  a  postcard  that  could  be  used  by  the  respondent  to  indicate  that  the  questionnaire  was  not 
appropriate  for  him/her. 

The  sample  was  drawn  from  the  membership  of  a  professional  aerospace  research  society  -  the  American 
Institute  for  Aeronautics  and  Astronautics.  Scientists  and  engineers  are  difficult  to  survey  for  two  reasons.  First, 
there  are  significant  problems  with  the  definition  of  scientists  and  engineers  (Citro  and  Kalton,  1989).  Second, 
typically  there  are  low  response  rates  in  surveys  of  this  group  (Citro  and  Kalton,  1989).  These  two  problems  were 
found  in  the  NASA  surveys. 

The  results  indicate  that  the  sponsor  letter  improved  response  rates  under  certain  conditions  described  in 
the  paper.  The  postcards  assisted  in  identifying  non-eligible  persons  particularly  when  they  accompanied  a  pre¬ 
survey  letter.  The  implications  for  survey  costs  are  discussed. 
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The  Impact  of  a  Sponsor  Letter  on  Mail  Survey  Response  Rates 

Survey  researchers  continually  search  for  techniques  to  improve  response  rates  and  reduce 
survey  costs  and  errors  In  this  paper,  we  describe  how  the  timing  of  a  letter  from  a  sponsoring 
government  agency  affected  the  return  rates  in  three  mail  surveys.  In  addition,  we  describe  how  a 
postcard  was  utilized  to  clarify  the  sample  in  the  surveys.  Together,  these  procedures  allowed  us  to  assess 
the  relative  impact  of  survey  design  changes  on  response  rates  and  survey  costs. 

Our  research  was  prompted  by  the  need  to  refine  our  mail  survey  procedures  for  repeated 
samples  from  a  specific  target  population  -  domestic  and  international  aerospace  scientists  and  engineers. 
In  the  past  year,  the  Center  for  Survey  Research  at  Indiana  University  has  conducted  six  surveys  for  the 
National  Aeronautics  and  Space  Administration  (NASA).  In  this  paper  we  present  some  of  the  results  from 
three  of  the  surveys  which  were  similar  in  content  and  target  population.  The  focus  of  the  research  is  the 
analysis  of  the  transfer  of  scientific  and  technical  information  (STI)  from  research  labs,  such  as  NASA’s,  to 
engineers  and  scientists  in  the  aerospace  industry  and  in  universities.  In  the  initial  phase  of  the  research 
project,  three  mail  surveys  of  aerospace  scientists  and  engineers  were  conducted  to  determine  their  use  of 
a  variety  of  information  sources  that  might  be  used  in  aerospace  research.  Subsequent  phases  of  the 
project  would  involve  surveying  additional  domestic  and  international  aerospace  scientists  and  engineers. 

We  began  our  research  project  using  the  Total  Design  Methods  described  by  Dillman  (1978).  In 
two  prior  surveys  of  professionals  conducted  at  the  Center  for  Survey  Research  these  procedures 
generated  response  rates  of  85  percent  (university  faculty)  and  73  percent  (lawyers  in  Indiana).  The  first 
two  surveys  in  the  NASA  project  generated  response  rates  of  less  than  60  percent.  Since  additional 
surveys  were  proposed  as  part  of  the  project,  we  decided  to  experiment  with  alternative  survey  procedures 
to  increase  response  rates.  The  procedures  tried  were:  1 )  sending  a  sponsor  letter  from  a  NASA  official  to 
the  non-respondents  in  the  first  two  surveys;  2)  sending  a  sponsor  letter  as  a  presurvey  letter  in  the  third 
survey;  and,  3)  including  a  postage-paid  return  postcard  with  the  sponsor  letters  that  would  allow  us  to 
define  more  dearly  the  sample.  In  addition,  we  hoped  that  the  results  of  these  changes  would  allow  us  to 
reduce  the  costs  of  the  tjture  surveys. 


Groves  (1989;  p.  49)  argues  that  costs  should  be  considered  as  a  factor  in  survey  design.  As  part 
of  our  research,  we  make  some  preliminary  estimates  of  the  costs  of  implementing  the  various  design 
features.  Our  goals  in  the  research  were  to  assess  the  effect  of  the  design  changes  on:  1 )  response  rates; 
2)  clarifying  the  appropriate  sample;  3)  the  relative  costs  of  each  design;  and  4)  the  demographic 
composition  of  the  respondents. 

Design  Effects  on  Mail  Survey  Response  Rates 

In  this  paper,  we  examine  the  impact  of  four  factors  that  have  been  found  to  increase  response 
rates  to  mail  surveys  -  viz,  university  sponsorship,  government  sponsorship,  number  of  contacts,  and 
presurvey  letters.  Fox  et  al.  (1988)  conducted  a  meta-analysis  of  nine  factors  thought  to  be  related  to 
improving  response  rates  to  mail  surveys.  In  their  research,  they  analyzed  148  articles  that  described 
various  mail  survey  design  procedures.  Among  the  factors  they  found  significantly  related  to  response 
rates  were  university  sponsorship  and  prenotification  letters. 

Earlier,  Heberfein  and  Baumgartner  (1978,  1981)  analyzed  the  impact  of  71  variables  used  in  98 
mail  survey  experiments  on  final  response  rates  to  mail  surveys.  They  found  that  ten  variables  including 
government  sponsorship  and  the  total  number  of  contacts,  could  predict  response  rates.  Goyder  (1982) 
replicated  the  research  of  Heberlein  and  Baumgartner  and  found  essentially  the  same  results.  Eichner  and 
Habermehl  (1981)  were  not  able  to  replicate  the  results  in  their  analysis  of  mail  surveys  conducted  in 
Europe. 

The  previous  research  has  demonstrated  that  response  rates  are  affected  by  a  variety  of 
characteristics  of  the  survey  procedures.  None,  though,  analyzed  the  results  in  terms  of  the  relative  costs 
of  the  procedures  and  the  improvements  in  response  rates.  Also,  the  research  cited  has  not  focused  on 
the  possibility  that  design  changes  could  differentially  alter  the  response  rates  among  various  demographic 
groups  in  the  sample.  We  suggest  that  procedures  designed  to  improve  response  rates  should 
simultaneously  recognize  the  costs  of  the  design  and  the  potential  of  differential  impact  on  non-response. 
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Surveying  Engineers  and  Scientists 

Typically  there  are  lew  response  rates  in  surveys  of  scientists  and  engineers  (Citro  ar.J  Kalton, 
1989)  One  of  the  reasons  often  given  to  explain  why  they  are  difficult  to  survey  is  because  of  the  problems 
in  defining  who  is  a  scientist  or  engineer  (Citro  and  Kalton,  1989).  We  expected  that  achieving  high 
response  rates  would  be  a  challenge  because  of  the  definition  problem  and  because  the  topic  of  the 
questionnaires  was  not  relevant  to  the  professional  activities  of  some  unknown  proportion  of  the  sample. 

The  focus  of  the  surveys  was  to  evaluate  the  information-seeking  habits  and  practices  of 
researchers  in  aerospace.  A  sample  of  aerospace  engineers  and  scientists  was  chosen  from  the 
membership  of  the  American  Institute  of  Aeronautics  and  Astronautics  (AIAA).  Most  researchers  in 
aerospace  belong  to  the  AIAA  because  it  is  their  primary  professional  research  society.  But,  many 
members  of  AIAA  are  no  longer  involved  in  aerospace  research  and  some  are  no  longer  in  the  aerospace 
industry.  Therefore,  many  of  the  members  of  the  sample  could  not  be  classified  as  aerospace  research 
engineers  or  scientists. 

Further,  we  assume  that  questionnaires  were  sent  to  some  AIAA  members  for  whom  the  questions 
had  no  relevance.  That  is,  many  in  the  sample  were  not  actively  engaged  in  aerospace  research  in  their 
professional  duties.  Therefore,  we  anticipated  that  the  response  rate  would  decrease  because  the  salience 
of  the  topic  was  expected  to  be  low  for  many  in  the  sample  (Hebertein  and  Baumgartner,  1978). 

The  problems  were  further  compounded  by  the  historically  low  response  rates  in  surveys  of 
scientists  and  engineers.  For  example,  the  National  Science  Foundation  conducted  a  survey  of  scientists 
and  engineers  in  1982.  In  this  survey,  the  sample  was  mailed  a  questionnaire,  a  postcard,  and  up  to  two 
additional  questionnaires  were  sent  to  the  non-respondents.  After  the  third  questionnaire,  a  telephone 
follow-up  was  attempted  with  the  remaining  non-respondents.  The  response  rate  for  the  survey  was  71 
percent.  (This  description  of  the  survey  came  from  Citro  and  Kalton;  pg  67.)  Citro  and  Kalton  (1989;  pg 
120)  present  additional  evidence  to  show  that  response  rates  from  this  group  have  decreased  during  the 


1980's. 
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Survey  Procedures 

The  first  two  NASA  surveys  were  conducted  following  the  procedures  described  in  Diilman  (1978). 
In  each  survey,  we  mailed  a  questionnaire,  a  postcard,  and  a  second  questionnaire  to  the  non¬ 
respondents.  In  the  first  survey,  we  mailed  a  third  questionnaire  to  the  non-respondents.  From  the  first 
survey,  we  received  about  58  percent  usable  questionnaires.  In  addition,  about  four  percent  of  the  sample 
returned  questionnaires  that  were  not  usable  (see  Table  1).  The  second  NASA  survey  was  staggered  one 
mailing  behind  the  first  survey  and  after  mailing  two  questionnaires  in  the  second  survey,  the  proportion  of 
usable  questionnaires  was  about  50  percent.  An  additional  five  percent  replied  but  did  not  provide  a 
usable  questionnaire. 

In  this  paper,  three  terms  are  used  to  describe  the  returns  from  the  surveys.  The  "response  rate’  is 
calculated  on  the  number  of  usable  responses  divided  by  the  total  sample  (RR  =  Usable/N).  The  "adjusted 
response  rate"  is  the  number  of  usable  response  divided  by  the  sample  minus  the  self-defined  ineligible 
persons  in  the  sample  (ARR  =  Usable/[N  -  ineligible]).  Respondents  identified  themselves  as  ineligible 
primarily  because  they  were  not  engaged  in  aerospace  research.  In  Table  1,  the  "other"  category  refers  to 
persons  who  were  retired,  moved  to  Europe,  deceased,  or  otherwise  ineligible  for  the  survey.  The  "return 
rate"  is  the  number  of  sample  persons  who  responded  in  any  form  divided  by  the  sample  size.  Comparing 
the  response  rate  against  both  the  adjusted  response  rate  and  the  return  rate  is  valuable  in  determining  the 
relative  costs  of  design  changes. 

The  results  in  Table  1  demonstrate  the  relatively  low  response  rates  in  the  two  surveys.  The 
proportion  of  usable  responses  was  much  less  than  had  been  our  experience  so  we  decided  to  continue 
the  survey,  primarily  to  increase  the  response  rate  but  also  to  determine  reasons  for  non-response  and  to 
improve  the  design  for  future  surveys. 
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Table  1 :  Detailed  Response  Outcomes  from  Two  Mail  Surveys. 


OUTCOMES 

NASA I1 

NASA2' 

Usable 

57.8 

50.0 

Refused 

0.8 

0.6 

Not  in  aerospace 

1.0 

1.0 

Other  reason 

1.9 

3.6 

Response  Rate 

58% 

50% 

Adjusted  Response  Rate 

59% 

52% 

Return  Rate 

61% 

55% 

1 .  Initial  mailing  and  postcard  sent  to  all  respondents;  up  to  two  additional  questionnaires  sent  to  non¬ 
respondents. 

2.  Initial  mailing  and  postcard  sent  to  all  respondents;  one  additional  questionnaire  sent  to  non¬ 
respondents. 


The  Implementation  of  Additional  Survey  Design  Features 

Two  additional  procedures  were  undertaken  to  increase  the  response  rate  and  to  establish  the 
relevance  of  the  survey  focus  for  the  remaining  persons  in  the  samples.  First,  a  letter  from  a  NASA  official 
was  sent  to  the  remaining  non-respondents.  The  letter  explained  the  importance  of  the  survey  and 
requested  cooperation  with  the  project.  All  aerospace  researchers  are  familiar  with  NASA,  so  it  was 
assumed  that  a  letter  from  a  NASA  Division  Director  would  induce  some  of  the  non-respondents  to 
complete  and  return  the  questionnaire.  The  impact  of  a  letter  from  a  government  sponsor  was  found 
previously  to  significantly  increase  response  rates  by  Heberlein  and  Baumgartner  (1978). 

In  order  to  identify  the  relevance  of  the  content  of  the  questionnaires,  a  return  postcard  was 
enclosed  with  the  letter.  This  postcard  allowed  the  recipient  to  check-off: 

1 .  Please  send  me  another  questionnaire 

2.  I  am  not  involved  in  aerospace  research 

3.  I  do  not  wish  to  participate  in  the  study 

4.  Please  send  me  a  copy  of  the  final  report 
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The  postcard  was  designed  to  help  determine  if  the  reason  for  the  previous  non-response  was  due 
to  the  content  of  the  questionnaire.  It  was  expected  that  this  procedure  would  allow  us  to  better  determine 
the  proportion  of  the  sample  that  did  not  respond  because  the  survey  focus  was  not  appropriate  to  them. 

At  the  same  time  as  the  sponsor  letters  were  sent,  the  third  survey  began.  Prior  to  mailing  the 
questionnaires,  a  presurvey  letter  from  a  NASA  official  was  sent.  This  letter  was  similar  to  the  follow-up 
letter  in  the  other  surveys.  Also  included  was  a  postcard  that  allowed  the  respondent  to  indicate  that  s/he 
is  not  involved  in  aerospace  research.  The  postcard  was  similar  to  that  sent  in  the  other  surveys  except 
that  it  did  not  have  a  check-off  to  send  another  questionnaire.  Two  weeks  were  allowed  for  the 
respondents  to  return  the  postcard.  At  that  time,  all  those  who  had  not  returned  the  postcard  were  sent 
questionnaires.  A  follow-up  postcard  and  a  second  questionnaire  to  non-respondents  were  also  sent. 


Table  2:  Detailed  Response  Outcomes  from  Three  Mail  Surveys;  Final  Results. 


OUTCOMES 

NASA  I1 

NASA22 

NASA3' 

Usable 

61.4 

55.6 

54.1 

Refused 

4.4 

5.3 

4.5 

Not  in  aerospace 

6.0 

7.5 

13.1 

Other  reason 

3.3 

3.7 

2.6 

Response  Rate 

61% 

56% 

54% 

Adjusted  Response  Rate 

67% 

63% 

64% 

Return  Rate 

75% 

72% 

74% 

1 .  Initial  mailing  and  postcard  sent  to  all  respondents;  up  to  two  additional  questionnaires  sent  to  non¬ 
respondents,  sponsor  letter  sent  last. 

2.  Initial  mailing  and  postcard  sent  to  all  respondents;  one  additional  questionnaire  sent  to  non¬ 
respondents,  sponsor  letter  sent  last. 

3.  Presurvey  sponsor  letter  sent  to  all,  mailing  and  postcard  to  all  who  did  not  indicate  not  applicable, 
one  additional  questionnaire  sent  to  non-respondents. 


The  differences  between  Tables  1  and  2  indicate  that  the  sponsor  letter  did  not  dramatically 
increase  response  rates.  The  Response  Rate  in  the  first  and  second  surveys  increased  by  about  4  and  6 
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percentage  points  respectively.  A  comparison  of  the  third  survey  with  the  results  of  the  second  survey 
prior  to  the  sponsor  letter  indicates  that  the  sponsor  letter  may  have  increased  the  Response  Rate  in  the 
third  survey  by  about  4  percentage  points.  The  response  rates  appear  to  be  in  the  range  currently 
experienced  in  surveys  of  engineers  and  scientists  {Citro  and  Kalton,  1989).  Since  both  government- 
sponsored  and  university-sponsored  surveys  were  found  to  increase  response  rates  to  mail  surveys,  it  is 
not  surprising  that  the  net  effect  of  the  sponsor  letter  was  small. 

There  is  some  evidence  that  the  postcard  was  effective  in  focusing  the  sample.  The  number  of 
persons  who  indicated  the  survey  was  not  appropriate  increased  substantially  after  the  postcard  was  sent 
in  the  first  two  surveys.  The  Adjusted  Return  Rates  increased  by  8  and  1 1  percentage  points  respectively  in 
the  two  surveys  that  received  the  postcard  in  the  last  mailing.  In  the  third  survey  about  13  percent 
indicated  the  topic  was  not  relevant.  About  70  percent  of  these  persons  indicated  so  prior  to  the  first 
mailing.  Overall,  it  seems  that  with  further  refinement  of  the  initial  letter  and  the  postcard  that  this 
procedure  can  be  used  to  refine  samples  where  there  is  a  problem  with  eligibility  for  the  survey. 

Some  further  cautions.  It  is  possible  that  at  least  three  other  factors  differentially  affected  the  response 
rates.  First,  the  form  of  the  questions  and  layout  of  the  questionnaires  was  similar  in  each  survey  but  the 
topics  in  each  was  a  little  different.  In  the  first  survey,  the  questions  focused  on  a  variety  of  government 
and  nongovernment  information  sources.  In  the  second  survey,  the  questions  focused  on  the  use  and 
availability  of  selected  government  and  foreign  technical  reports.  In  the  third  survey,  bibliographic  sources 
of  information  about  technical  reports  were  researched.  The  data  from  the  third  survey  indicate  that  very 
few  of  the  respondents  were  familiar  with  most  of  the  bibliographic  sources  and  we  suspect  this  lack  of 
knowledge  influenced  the  response  rate.  Since  the  results  are  similar  across  surveys,  if  these  factors 
affected  response,  they  probably  did  so  independent  of  the  letter. 

Second,  the  questionnaires  in  each  survey  were  different  colors.  The  first  survey  questionnaires 
were  blue  or  green.  Both  colors  had  about  equal  response  rates.  The  second  questionnaire  was  a  light 
brown  and  the  third  questionnaire  was  white.  Fox  et  al  (1988)  found  significant  increases  in  response  rates 
associated  with  green  vs  white  questionnaires.  Another  possible  factor  that  affected  response  rates  was 
the  time  each  survey  was  in  the  field.  The  first  survey  started  in  early  summer,  the  second  started  in  early 
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July,  and  the  third  survey  began  in  early  fall.  We  suspect  that  summer  vacations  and  household  migration 
affected  the  number  of  returns. 

The  results  from  these  surveys  and  design  experiments  along  with  the  results  from  other  surveys  of 
scientists  and  engineers  indicate  that  it  would  be  difficult  to  increase  response  rates  above  the  mid-€0 
percent  levels.  Therefore,  the  procedures  for  future  mail  surveys  can  be  changed  to  reflect  the  relative 
costs  and  the  overall  impact  on  nonresponse  bias.  That  is,  if  there  are  substantial  differences  in  the  costs 
of  each  set  of  procedures,  it  would  appear  to  be  best  to  use  the  least  costly  design  so  that  more 
questionnaires  could  be  returned.  Further,  if  a  procedure  affects  nonresponse  bias  (see  Groves,  1989;  pg 
40-41  for  a  definition  of  non-response  bias)  it  should  be  evaluated  carefully  before  being  implemented.  In 
the  following  sections,  each  of  these  concerns  is  addressed. 

Comparative  Costs  of  Each  Design 

Table  3  contains  data  on  the  estimated  costs  of  each  design  and  the  relative  value  of  each.  These 
costs  reflect  only  the  wages  paid  to  the  staff  to  prepare  the  mailings  and  the  postage  costs.  The  wages  are 
further  limited  to  only  the  time  required  for  printing  individually  addressed  letters  on  a  laser  printer,  signing 
the  letters,  printing  envelopes  and  postcards  on  a  dot-matrix  printer  with  an  envelope  feeder,  preparing  the 
mailing  packages,  and  checking  the  accuracy  of  the  names,  addresses,  and  control  numbers.  These  costs 
were  calculated  for  the  first  survey  and  the  costs  for  the  other  two  survey  costs  were  standardized  on  the 
first  survey. 

The  data  in  Table  3  permit  an  evaluation  of  both  the  sponsor  letter  and  the  postcard.  From  these 
data,  it  appears  that  the  sponsor  letter  did  not  produce  responses  equal  to  their  costs.  For  example,  the 
costs  of  two  or  three  mailings  without  the  sponsor  letter  was  less  than  $3.00  per  response.  When  the 
sponsor  letter  was  sent  as  either  a  follow-up  after  two  or  three  mailings  or  as  a  presurvey  letter,  the  costs 
increased  substantially.  The  third  survey  and  the  first  survey  had  an  equivalent  number  of  contacts.  The 
results  further  indicate  that  the  presurvey  letter  is  relatively  costly  and  does  not  produce  as  many  returns 
(at  a  lower  unit  cost)  as  mailing  a  third  questionnaire. 
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Table  3:  Costs  per  Response  With  Various  Mail  Survey  Design  Procedures. 


PROCEDURES  AND  COSTS 

NASA11 

NASA22 

NASA33 

Cost  per  response  prior 

to  sponsor  letter 

$2.92 

$2.96 

Response  Rate 

58% 

50% 

Adjusted  Response  Rate 

59% 

52% 

Return  Rate 

61% 

55% 

Cost  per  response  - 

sponsor  letter  only 

$8.41 

$6.04 

Cost  per  response  with 

a  sponsor  letter 

$3.21 

$3.27 

$3.12 

Response  Rate 

61% 

56% 

54% 

Adjusted  Response  Rate 

67% 

63% 

64% 

Return  Rate 

75% 

72% 

74% 

1  Initial  mailing  and  postcard  sent  to  ail  respondents;  up  to  two  additional  questionnaires  sent  to  non¬ 
respondents.  sponsor  letter  sent  last. 

2.  Initial  mailing  and  postcard  sent  to  all  respondents;  one  additional  questionnaire  sent  to  non¬ 
respondents,  sponsor  letter  sent  last. 

3.  Presurvey  sponsor  letter,  mailing  and  postcard  to  all  who  did  not  indicate  not  applicable,  one 
additional  questionnaire  sent  to  non-respondents. 


If  the  Adjusted  Response  Rate  is  used  to  evaluate  the  procedures,  then  the  postcard  does  seem  to 
be  cost-effective.  In  the  analysis  of  Table  2,  it  was  demonstrated  that  the  postcard  helped  to  refine  the 
sample  by  improving  the  Adjusted  Response  Rate.  Each  of  the  surveys  has  about  the  same  Adjusted 
Response  Rate  but  the  third  survey  cost  somewhat  less  than  the  other  two. 

Overall,  the  result*  indicate  that  a  sponsor  letter  as  a  follow-up  to  a  survey  is  not  cost  effective; 
especially  since  the  cost  per  return  of  the  extra  mailing  was  over  $8.00  and  $6.00  in  the  two  surveys.  The 
data  did  not  support  our  assumption  that  a  presurvey  letter  with  a  postcard  would  reduce  costs.  We 
assumed  that  ineligible  or  non-cooperative  respondents  would  be  eliminated  and  this  would  reduce  costs 
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because  of  fewer  mailings.  The  postcard  was  cost  effective  in  clarifying  the  sample;  especially  when  it  was 
sent  with  a  presurvey  letter. 

Impact  of  the  Procedures  on  Non-Responae  Bias 

The  list  of  AIM  members  that  comprises  the  frame  for  the  sample  includes  a  number  of 
characteristics  that  were  also  measured  in  the  questionnaire.  Two  characteristics  -  education  and 
government/non-government  employment  -  are  relevant  to  the  focus  of  the  research.  Ideally,  there  should 
be  no  difference  between  the  distributions  of  these  characteristics  in  the  frame  and  in  the  data  from  each  of 
the  surveys.  We  did  suspect  that  the  sample  would  be  biased  on  these  two  characteristics.  First,  scientists 
and  engineers  with  advanced  degrees  would  be  more  likely  to  be  engaged  in  aerospace  research  and 
therefore  more  likely  to  find  the  questionnaires  to  be  relevant.  Second,  we  expected  that  those  employed 
in  government  agencies  would  be  more  likely  to  return  the  questionnaires  because  they  were  more  likely  to 
be  involved  in  aerospace  research. 

The  data  in  Table  4  support  our  hypotheses.  First,  there  is  clear  evidence  that  the  a  larger 
proportion  of  respondents  to  each  of  the  surveys  has  advanced  degrees  than  in  the  sample  frame  in 
general.  In  the  sample  frame  only  57  percent  have  advanced  degrees  but  in  the  surveys  at  least  66  percent 
reported  they  have  a  master's  degree  or  a  doctorate. 

The  academic  portion  of  the  survey  was  conducted  separately  so  those  classified  as  non¬ 
government  in  employment  question  are  primarily  working  in  industry.  The  relatively  higher  proportion  of 
government  agency  employees  may  reflect  differences  in  professional  duties.  That  is,  we  expect  that 
members  of  the  AIM  who  are  employed  in  government  settings  are  more  likely  to  be  engaged  in 
aerospace  research.  Many  of  the  professional  duties  of  members  of  the  AIM  who  are  in  industry,  e.g., 
sales  and  service,  are  not  found  in  government. 
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Table  4:  Characteristics  of  the  Sample  Frame  and  the  Respondents  to  Three  Surveys  of 


Members  of  AAIA.  (percents) 

SAMPLE 

FRAME 

CHARACTERISTICS 


Education 

Bachelor's  36 

Master’s  36 

Doctorate  21 

Other  7 

Employment 

Non-Government  80 

Government  20 


NASA1 

NASA2 

NASA3 

31 

31 

29 

43 

45 

44 

23 

22 

24 

3 

2 

3 

74 

74 

77 

26 

26 

23 

These  data  may  indicate  a  non-response  bias  but  they  can  also  be  interpreted  in  other  ways.  First, 
we  are  not  certain  that  the  distribution  of  responses  to  any  questions  on  the  survey  would  be  different  if  we 
had  received  questionnaires  from  all  persons  in  the  sample.  Second,  the  non-respondents  may  not  be 
actively  engaged  in  aerospace  research.  If  so,  then  if  the  non-respondents  did  return  questionnaires,  it 
would  create  another  form  of  bias.  That  is,  they  would  provide  answers  even  though  the  questions  were 
not  relevant.  It  is  possible  that  questionnaires  were  received  from  almost  all  persons  to  whom  the  focus  of 
the  research  was  relevant.  Overall,  it  is  not  clear  what  impact  the  procedures  had  on  errors  associated  with 
non-response  to  the  surveys. 

Summary 

In  this  paper  we  described  the  impact  of  two  interventions  (a  sponsor  letter  and  a  postcard)  on 
improvements  to  response  rates  and  the  clarification  of  a  sample.  We  evaluated  the  effectiveness  by  an 
increase  in  the  response  rates,  the  relative  costs  of  the  procedures,  and  the  impact  of  the  procedures  on 
the  characteristics  of  the  respondents.  The  results  indicate  that  the  sponsor  letter  had  little  impact  on  the 
results.  The  increased  cost  of  mailing  a  separate  letter  from  a  sponsor  did  not  result  in  as  many  responses 
as  would  another  mailing  of  the  questionnaire. 


The  postcard  appears  to  be  an  effective  means  of  clarifying  the  appropriate  sample  when  there  is  a 
sample  frame  that  may  contain  persons  who  are  not  eligible  for  the  survey.  The  postcard  was  especially 
effective  when  it  was  included  with  a  presurvey  letter.  In  addition,  the  postcard  sent  with  the  presurvey 
letter  was  cost  effective  when  compared  to  mailing  the  postcard  after  the  survey  had  begun. 

None  of  the  three  mail  survey  designs  produced  differences  in  the  characteristics  of  the  sample. 
Likewise,  all  designs  showed  evidence  of  non-response  bias.  It  is  not  possible  to  estimate  the  impact  of 
the  nonresponse  bias  but  it  can  be  argued  that  the  non-respondents  may  not  have  been  eligible  for  the 
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